Prior to the development of digital imaging systems, cameras were based on a lens and a film-based a photographic emulsion located at the focal plane of the lens. An optical image of a scene would be projected onto the emulsion, which would permanently record the image via a chemical-based process. The advent of digital imaging has enabled advances in the way that an image of a scene can be recorded and viewed, however. In particular, a modern camera forms an image of a scene by temporarily digitally recording the optical image of the scene using an electronic-sensor array located at the focal plane of the imaging lens. A sensor array (a.k.a., focal-plane array) normally comprises a large two-dimensional array of optoelectronic detector pixels, such as charge-coupled device (CCD) elements, photodetectors, etc. The sensor array generates a digital image-data set based on the sub-image formed on its recording surface during image capture.
As digital-imaging technology has matured, sensor arrays containing ever-larger pixel counts have been developed, since an imaging system having high pixel count offers many advantages for viewing an output image. For example, an image of a total scene can be provided at improved image quality, while the improved resolution also offers the potential for enlarging the view of sub-regions of the scene to enable their examination in greater detail.
Pixel count is a basic measure of image quality and is commonly specified by the number of mega pixels an image contains. In most cases, the sensor elements are included in a single sensor array. Since the relative position of the sensor elements in a single array is known a priori and remains fixed throughout the imaging process, the use of a single sensor array facilitates the image processing required to convert raw pixel data from the sensor array into an output image in a reasonable amount of time. The total number of pixels is limited by the size of each sensor element and the practical limit for the size of the substrate on which they can be formed. A typical mobile phone or digital camera has a pixel count within the range of 8-40 million.
In many cases, it is desirable to expand the number of image pixels beyond what can be conveniently derived by imaging a scene onto a single sensor array. In the prior art, this has typically been achieved by aggregating multiple digital sub-images, each provided by a different sensor array, into a composite image that has large pixel count while maintaining high pixel density within each sub-image region. This offers performance advantages over single-sensor-array cameras, such as a wide field-of-view combined with high angular resolution, high-speed data readout, and lower cost-per-pixel compared to systems with one continuous focal-plane-array.
Panoramic imaging is an example of an application wherein multiple low-pixel-count images are combined to assemble a high-pixel-count image. Most often, a panoramic image is developed by taking a series of images from a single camera while the camera is panned and tilted during acquisition of the series. Alternatively, camera systems having multiple sensor arrays are also sometimes used.
In some cases, array cameras employing multiple single-sensor-array microcameras are used in panoramic and non-panoramic imaging applications. In such systems, each microcamera provides output data based on a different portion of a scene to a common image-aggregation processor that combines the data into a composite image of the entire scene.
Unfortunately, assembling large composite images from multiple smaller sub-images is very computationally intensive due to the geometrical and radiometric processing of the sub-images that is required to stitch the sub-images together. Further, when the sub-images are often taken at different times, the illumination of the scene can change or there can be motion artifacts associated with objects moving within the field-of-view. Still further, the responsivity of different sensor arrays can be different giving rise to variations in contrast, brightness, etc. As a result, algorithms that compare neighboring images are required in order to mitigate seams between sub-images due to these variations. In addition, distortion, pointing, and non-linearity corrections must be applied to the sub-images. Once this extensive processing is complete, a single image file having very high-pixel-count can be obtained (typically, tens of mega pixels to tens of gigapixels in size).
Such extensive processing imposes a severe time constraint, however, which has historically precluded using multiple sensor arrays for video-rate capture of high-resolution, high-pixel-count imagery. To date, therefore, high-definition video streams have been principally limited to single-sensor-array camera acquisition. As a result, in video-rate applications, numerous separately controlled cameras are typically used to capture a complete scene, where each camera provides only a small-area view of a portion of the scene. For example, a sports broadcast normally relies on the use of many different cameras that are strategically positioned and oriented throughout an arena or stadium. Each camera requires its own camera operator and the multiple camera views must be continuously analyzed in real time by a director who chooses which one camera view is broadcast. In addition to giving rise to inordinate capital and operational expense, such an approach limits the “richness” of the viewing experience.
The need for an imaging system that can provide high-resolution imagery of an entire scene at the same time remains, as yet, unmet.